AMENDMENTS TO THE CLAIMS 



1. (Currently amended) A method for automating the optimization of search results 
displayed in a search Web page, the method comprising: 

providing search results to users of a search engine; 

collecting data that represents a performance of a search result, the data originating from 
at least one of a plurality of sources of performance data that includes implicit data collected by 
quantifying the user's interactions with the search results ; 

normalizing the collected data that describes the user's interactions with the search results 
in accordance with a relative importance of the source of the data; 

comparing the normalized performance data to an expected performance data for the 
search result; 

diagnosing at least one possible cause for an underperforming search result from the 
results of the comparison between the normalized performance data and the expected 
performance data; and 

adjusting an operation of a search engine that produced the search result in accordance 
with the diagnosis to improve the search result performance, the adjustment operative to give 
greater significance to collected performance data from at least one of the sources when 
normalizing the data . 

2. (Original) The method of Claim 1, wherein the at least one of the plurality of 
sources of performance data includes one of an implicit performance data, an explicit 
performance data, a human-judged performance data, a relevance verification data, and a sample 
test data. 
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3. (Currently amended) The method of Claim 1, wherein normalizing the collected 
data in accordance with a relative importance of the source of the data includes giving greater 
weight to the collected data from the more important sources that describes the user's interactions 
with the search results and combining the data to reflect the relative importance of the source 
from which the data originated. 

4. (Previously presented) The method of Claim 2, wherein comparing the 
normalized performance data to the expected performance data includes determining that the 
search result is underperforming when the normalized performance is below a quantified 
threshold below the expected performance. 

5. (Previously presented) The method of Claim 2, wherein the most important 
source of data is implicit performance data, and normalizing the collected data includes giving 
implicit performance data greater weight when combining the data. 

6. (Original) The method of Claim 2, wherein implicit performance data is 
automatically captured when a user interacts with the search result. 

7. (Original) The method of Claim 6, wherein implicit performance data includes at 
least one of whether the user clicked on the result, a location of the result when the user clicked 
the result, and a length of time that the user browsed the result. 

8. (Original) The method of Claim 7, wherein implicit performance data further 
identifies an operation that the user performed on the result, including at least one of editing, 
e-mailing, printing, bookmarking, and copying. 
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9. (Original) The method of Claim 6, wherein implicit performance data includes 
captured data that has been aggregated across multiple interactions with the search result. 

10. (Original) The method of Claim 6, wherein implicit performance data includes 
captured data that has been aggregated across multiple users interacting with the search result. 

11. (Original) The method of Claim 2, wherein explicit performance data includes 
data obtained from user response to a search engine operator- generated inquiry about the search 
result, wherein the inquiry includes one of an on-line inquiry and a telephone inquiry. 

12. (Original) The method of Claim 2, wherein human-judged performance data 
includes data obtained from a human evaluation of the search result. 

13. (Original) The method of Claim 2, wherein relevance verification data includes 
data obtained from a verification test of the search result's relevance to verify whether a search 
result having a known relevance is still included in the search result generated by the search 
engine. 

14. (Original) The method of Claim 2, wherein sample test data includes data 
obtained from a test of the search result's relevance performed on a sample of a subset of users. 

15. (Original) The method of Claim 2, wherein diagnosing at least one possible cause 
for an underperforming search result includes considering at least one of whether the search 
result is no longer valid, whether the search result appears in a poor location, whether a search 
term that generated the search result is easily misspelled, whether the search term is too broad to 
generate a meaningful result, and whether a search for the search term should be constrained to a 
specific resource. 
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16. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes modifying the search engine's search schema to change the search 
result generated for the search term, including at least one of reranking, removing, and replacing 
the search results. 

17. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes modifying the search engine's search schema to augment a 
presentation of the search results generated for the search term, including at least one of 
highlighting, animating, enlarging, and repositioning an appearance of the search result on a 
search results web page. 

18. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes increasing the search engine's spellchecker tolerance. 

19. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes prompting the user to one of clarify or narrow the search term with 
an additional user input. 

20. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes temporarily adjusting the operation of the search engine, and 
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determining whether the adjustments have actually improved the search result performance 
before permanently adjusting the operation of the search engine. 

21. (Original) The method of Claim 2, wherein adjusting an operation of a search 
engine that produced the search result in accordance with the diagnosis to improve the search 
result performance, includes adjusting the operation of the search engine in real time. 

22. (Original) The method of Claim 21, wherein adjusting the operation of the search 
engine in real time includes intercepting the search result generated by the search engine and 
modifying the search result before the search engine displays the search result to the user. 

23. (Currently amended) An automated search result optimization system 
implemented in a computer that provides search results to a user, comprising: 

an input data representing a performance of a search result generated by a search engine 
that is executed on the computer for a search term, the input data originating from a plurality of 
sources; 

a data collection process to collect the input performance data from the plurality of 
sources wherein the input data collected includes implicit data that describes the user's 
interactions with the search result ; 

a diagnostic process to compare a performance of the search result, as represented by the 
collected performance data, to an expected performance of the search result; and 

an adjustment process to generate an output data representing an action to automatically 
adjust an operation of the search engine whenever the search result performance diverges a 
quantified threshold from the expected performance, wherein the adjusted operation of the search 
engine is to improve the performance of the search result ; and 
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wherein the adjustment gives greater significance to performance data from one of the 
sources based on the implicit data that describes the user's interactions with the search result . 

24. (Original) The system of Claim 23, wherein the at least one of the plurality of 
sources of input performance data includes one of an implicit performance data, an explicit 
performance data, a human-judged performance data, a relevance verification data, and a sample 
test data. 

25. (Original) The system of Claim 23, wherein the data collection process includes a 
normalizing process to normalize the collected data in accordance with a relative importance of 
the source of the data, and further wherein normalizing includes giving greater weight to the data 
from the more important sources and combining the data to reflect the relative importance of the 
source from which the data originated. 

26. (Previously presented) The system of Claim 25, wherein the diagnostic process 
compares the performance of the search result as represented by the normalized collected 
performance data to an expected performance of the search result; and further, wherein the 
comparison is unfavorable when the performance is lower than a quantified threshold value 
below the expected performance. 

27. (Previously presented) The system of Claim 25, wherein the most important 
source of data is implicit performance data, and normalizing the collected data includes giving 
implicit performance data greater weight when combining the data. 

28. (Original) The system of Claim 27, wherein implicit performance data is 
automatically captured when a user interacts with the search result. 
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29. (Original) The system of Claim 28, wherein implicit performance data includes at 
least one of whether the user clicked on the result, a location of the result when the user clicked 
the result, and a length of time that the user browsed the result. 

30. (Original) The system of Claim 29, wherein implicit performance data further 
includes data representing an identification of an operation that the user performed on the result, 
including at least one of editing, e-mailing, printing, bookmarking, and copying. 

31. (Original) The system of Claim 28, wherein implicit performance data includes 
captured data that has been aggregated across multiple interactions with the search result. 

32. (Original) The system of Claim 28, wherein implicit performance data includes 
captured data that has been aggregated across multiple users interacting with the search result. 

33. (Original) The system of Claim 24, wherein explicit performance data includes 
data obtained from user response to a search engine operator-generated user satisfaction survey 
about the search result. 

34. (Original) The system of Claim 24, wherein human-judged performance data 
includes data obtained from a human evaluation of the search result. 

35. (Original) The system of Claim 24, wherein relevance verification data includes 
data obtained from a verification test of the search result's relevance to verify whether a search 
result having a known relevance is still included in the search result generated by the search 
engine. 

36. (Original) The system of Claim 24, wherein sample test data includes data 
obtained from a test of the search result's relevance performed on a sample of a subset of users. 
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37. (Original) The system of Claim 24, wherein the diagnostic process further 
includes diagnosing at least one possible reason why the performance of the search result 
compares unfavorably to the expected performance, including considering at least one of 
whether the search result is no longer valid, whether the search result appears in a poor location, 
whether a search term that generated the search result is easily misspelled, whether the search 
term is too broad to generate a meaningful result, and whether a search for the search term 
should be constrained to a specific resource. 

38. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to modify the search engine's search 
schema, wherein the modified search schema changes the way the search engine generates a 
search result for a search term, including at least one of reranking, removing, and replacing the 
search result. 

39. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to modify the search engine's search 
schema, wherein the modified search schema augments the way the search engine presents the 
search result for a search term, including at least one of highlighting, animating, enlarging, and 
repositioning an appearance of the search result on a search result Web page. 

40. (Currently amended) The system of Claim 23, wherein the adjustment process to 
generate an output data representing an action to automatically adjust an operation of the search 
engine, includes generating output data that represents an action to increase the search engine's 
spellchecker tolerance without changing the spelling or creating a variation of the search term . 
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41. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to prompt the user to one of clarify or 
narrow the search term with an additional user input. 

42. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to temporarily adjust the operation of 
the search engine, and to further determine whether the temporary adjustment has actually 
improved the search result performance before generating an action to permanently adjust the 
operation of the search engine. 

43. (Original) The system of Claim 23, wherein the adjustment process to generate 
an output data representing an action to automatically adjust an operation of the search engine, 
includes generating output data that represents an action to adjust the operation of the search 
engine in real time. 

44. (Currently amended) The system of Claim 43 herein wherein the action to adjust 
the operation of the search engine in real time includes an action to intercept the search result 
generated by the search engine and to further modify the search result before the search engine 
displays the search result to the user. 

45. (Currently amended) One or more computer-accessible media having instructions 
stored on the media for facilitating the automated optimization of a search result in a search 
result user interface, the instructions comprising: 

providing search results to users of a search engine; 
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collecting data that represents a performance of a search result from at least one of a 
plurality of sources of performance data that includes implicit data collected by quantifying the 
user's interactions with the search results ; 

normalizing the collected data that describes the user's interactions with the search results 
in accordance with a relative importance of the source of the data; 

comparing the normalized performance data to an expected performance data for the 
search result; 

diagnosing at least one possible cause for an underperforming search result based on the 
comparison between the normalized performance data and the expected performance data; and 

adjusting an operation of a search engine that produced the search result in accordance 
with the diagnosis to improve the search result performance, the adjustment operative to give 
greater significance to collected performance data from at least one of the sources when 
normalizing the data . 

46. (Original) The computer-accessible media of Claim 45, wherein the at least one 
of the plurality of sources of performance data includes one of an implicit performance data, an 
explicit performance data, a human-judged performance data, a relevance verification data, and a 
sample test data. 

47. (Currently amended) The computer- accessible media of Claim 45, wherein the 
instruction to normalize the collected data that describes the user's interactions with the search 
results in accordance with a relative importance of the source of the data includes instructions to 
give greater weight to the data from the more important sources and to combine the data to 
reflect the relative importance of the source from which the data originated. 
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48. (Previously presented) The computer- accessible media of Claim 45, wherein the 
instruction to compare the normalized performance data to the expected performance data 
includes an instruction to determine that the search result is underperforming when the 
normalized performance is below a quantified threshold value below the expected performance. 

49. (Previously presented) The computer-accessible media of Claim 47, wherein the 
most important source of data is implicit performance data, and the instruction to normalize the 
collected data includes an instruction to give implicit performance data greater weight when 
combining the data. 

50. (Original) The computer-accessible media of Claim 46, wherein implicit 
performance data is data that is automatically captured by a search engine when a user interacts 
with a search result. 

51. (Original) The computer-accessible media of Claim 50, wherein implicit 
performance data includes data that represents at least one of whether the user clicked on the 
result, a location of the result when the user clicked the result, and a length of time that the user 
browsed the result. 

52. (Original) The computer-accessible media of Claim 50, wherein implicit 
performance data further identifies an operation that the user performed on the result, including 
at least one of editing, e-mailing, printing, bookmarking, and copying. 

53. (Original) The computer-accessible media of Claim 50, wherein implicit 
performance data includes captured data that has been aggregated across multiple interactions 
with the search result. 
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54. (Original) The computer-accessible media of Claim 50, wherein implicit 
performance data includes captured data that has been aggregated across multiple users 
interacting with the search result. 

55. (Original) The computer- accessible media of Claim 46, wherein explicit 
performance data includes data obtained from user response to a search engine operator- 
generated inquiry about the search result, wherein the inquiry includes one of an on-line inquiry 
and a telephone inquiry. 

56. (Original) The computer-accessible media of Claim 46, wherein human-judged 
performance data includes data obtained from a human evaluation of the search result. 

57. (Original) The computer-accessible media of Claim 46, wherein relevance 
verification data includes data obtained from a verification test of the search result's relevance to 
verify whether a search result having a known relevance is still included in the search result 
generated by the search engine. 

58. (Original) The computer-accessible media of Claim 46, wherein sample test data 
includes data obtained from a test of the search result's relevance performed on a sample of a 
subset of users. 

59. (Currently amended) The computer-accessible media of Claim 45, wherein the 
instruction to diagnose at least one possible reason why the search result performance compares 



search result includes generating an instruction to determine at least one of whether the search 
result is no longer valid, whether the search result appears in a poor location[[,]] and whether a 
search term that generated the search result is easily misspelled , whether the search term is too 





diagnosing at least one possible for an underperforming 
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broad to generate a meaningful result, and whether a search for the search term should be 
constrained to a specific resource . 

60. (Original) The computer- accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to modify the search engine's search schema to change the 
search result generated for the search term, wherein to change the search result includes at least 
one of to rerank, remove, and replace the search result. 

61. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to modify the search engine's search schema to augment a 
presentation of the search result generated for the search term, wherein to augment the 
presentation includes at least one of to highlight, animate, enlarge, and reposition a display of the 
search result on a search result Web page. 

62. (Original) The computer- accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to increase the search engine's spellchecker tolerance. 

63. (Original) The computer- accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to prompt the user to one of clarify or narrow the search term 
with an additional user input. 

64. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
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diagnosis, includes an instruction to temporarily adjust the operation of the search engine, and an 
instruction to further determine whether the adjustments have actually improved the search result 
performance before performing an instruction to permanently adjust the operation of the search 
engine. 

65. (Original) The computer-accessible media of Claim 45, wherein the instruction to 
adjust the operation of the search engine that produced the search result in accordance with the 
diagnosis, includes an instruction to adjust the operation of the search engine in real time. 

66. (Original) The computer-accessible media of Claim 65, wherein the instruction to 
adjust the operation of the search engine in real time includes an instruction to intercept the 
search result generated by the search engine and a further instruction to modify the search result 
before the search engine displays the search result to the user. 
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